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Abstract:- The process of restoring missing areas of an image is referred to as "image inpainting." It is a 
significant challenge in the field of computer vision and a crucial feature that is utilized in a wide variety of image 
and graphics programs. Although image inpainting, also known as the art of repairing old and worn images, has 
been around for a couple of years, it has recently gained even more popularity due to recent developments in 
image processing techniques. Image inpainting can be thought of as the art of restoring old and worn images. 
Automatic image inpainting has become an important and challenging area of research in image processing due 
to the advancement of tools for image processing and the flexibility of digital image editing. Automatic image 
inpainting has found important applications in computer vision and is also becoming an important image 
processing application. Due to its high significance and effectiveness in a variety of image processing 
applications such as, for example, object removal, image restoration, manipulation, re-targeting, compositing, 
and image-based rendering, researchers have studied the image inpainting problem intensively over several 
decades. The process of eliminating or filling in a missing area in an image is referred to as "image inpainting,” It 
is defined as in-depth knowledge of the image details in terms of its structure and texture. It is considered to be 
one of the most challenging subjects in image processing. This article presents a survey of most image 
inpainting techniques and summarizes them, along with comparisons that include the benefits and drawbacks of 
each method. These comparisons and summaries can assist researchers in evaluating their own proposed 
techniques against existing ones. 


Keywords: Image inpainting, Texture synthesis methods, Exemplar-based texture synthesis, Generative 
Adversarial Network, Convolutional Neural Network. 


1 Introd uction information in an original image, or removes specific 
information in an image by using undamaged information 


from the original image while ensuring that the quality of 


The image inpainting operation is the practical ; : 
image and its natural effects are not damaged. The 


use of the computer to fill and repair broken areas of ; pimenie ac ; 
purpose of image inpainting procedure isn’t to restore on 


the basis of original image source, but rather to fix and 
reconstruct the images that are approximate to original 


source image in order to finally achieve visualization 
rationality. Moreover, an image inpainting algorithm is a 
kind of image processing method that repairs missing 
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image source [1]. The image inpainting procedure, as a 
common image editing procedure, aims to fill the missing 
or masked areas in original images with reasonable, 
plausible and synthesized contents. Image inpainting 
algorithms have been widely used in several research 
fields, including computer science and _ technology, 
electronic systems, machine learning, and computer 
vision. Traditional image inpainting algorithms have 
undergone nearly two decades of development. From the 
initial repairing algorithms utilizing a partial differential 
formula to image repairing algorithms based on image 
samples, existing information is mined and utilized from 
the image to repair the changes to a given area. The main 
objective of image inpainting is to produce completed 
recovered images in an unnoticeable manner to the 
human visual system. Image inpainting has various 


1(a) Image Restoration 


1(b) Object Removal 


applications, such as image restoration, image editing, 
removal of the unwanted object image denoising, etc. 
furthermore the digital Image In-painting has various 
applications such as restoration of damaged old printing 
and old photographs, error recovery of images and 
videos, multimedia editing (computer assisted), 
transmission loss and replacing large regions in an image 
or video for privacy protection. The concept of image 
inpainting existed very long year’s back and from the 
birth of computer vision, researchers are looking for a 
way to carry out this process automatically. Image 
Inpainting restructure the damaged region or mislaid 
parts in an image utilizing spatial information of 
neighboring region. Furthermore. Some _ different 
applications of image inpainting are shown in Fig. 1. 


Figure 1. Different application of Image inpainting 


Some of the inpainting approaches displayed in 
figure 2 have been great successes in nearest neighbor 
searching, which is the process of synthesizing images 
using patches or pixels. However, one of the difficulties 
associated with inpainting is preserving a realistic 
structure and texture in the final image. Traditional 
approaches, for instance, may try to fill in missing pixels 
by employing image patches taken from already existing 
regions. Alternatively, they will employ the diffusion 
process to transmit pixels into a hole region from regions 
with high pixel similarity. Although these algorithms can 
produce vibrant textures for background inpainting, they 
frequently fail to capture high-level semantics, which 
results in images that are not realistic and contain 
repeated patterns. Furthermore, these procedures do not 
produce plausible results. These traditional methods were 
suitable for completing small missing regions such as a 
crack in an image. However, with the evolution of digital 
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images, the task of inpainting became more complex as 
larger regions now are required to be filled regardless of 
its size or location across the image. Unfortunately, 
image inpainting traditional methods could not handle 
filling larger complex regions. 

Removing objects from images using image 
inpainting can reach improved performance in the future, 
but when the image editors hide traces using 
sophisticated techniques, the detection of forgery and the 
inpainting of image become difficult. For that reason, 
almost all detection approaches attempt to handle this by 
detecting the abnormalities of similarity between blocks 
of the image that can be affected during the 
postprocessing operation. This work summarizes 
different methods for image inpainting using different 
techniques including sequential-based, CNN-based or 
GAN-based methods. 
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Figure 2 shows a hierarchical representation of the various categories of inpainting in their respective groups 


The paper's main contribution is exploring the 
various types of traditional and deep learning-based 
image inpainting techniques and their effects. Research 
difficulties and suggestions could also be mentioned. 


2. Traditional Image Inpainting 
Algorithms 


The majority of well-known traditional image 
inpainting methods that do not involve deep learning 
methodologies are discussed in this section. These 
algorithms can be broken down even further into the five 
sub-categories shown in Figure 2. 

2.1 Diffusion-Based Inpainting: 


For physical phenomena like heat transfer in 
physical structures, diffusion is derived from spreading 
local information with smoothness constraints. PDEs can 
formalize these processes, and PDE-based regularization 
is utilized to perform diffusion. By seamlessly 
propagating local image structures from the hole's 
exterior to its interior, diffusion-based inpainting 
"imitates" competent painting restorers' actions. In 2000, 
Bertalmio et al. [2] were the first to use regularization or 
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The remaining paper is organized as follows: 
Section 2 explains the traditional image inpainting 
techniques, Section 3 describes the deep learning-based 
algorithms, Section 4 highlights the research problems 
and ideas, and Section 5 concludes the survey. 


diffusion for image inpainting. Diffusion-based 
inpainting is the process of filling an area by diffusing 
information from a known area into the target area . 
Diffusion algorithms are built on vibrational mathematics 
and partial differential equation (PDE) methods. The 
diffusion-based inpainting algorithm produces precise 
results when filling in non-textured or misplaced areas. 
Figure 3 depicts how the PDE approach propagates (or 
diffuses) neighboring data into the center of the hole. The 
illustration also shows the regions of well-known 
(source) and unknown (hole). Massive textured areas are 
filled with a fuzzy appearance due to the diffusion 
process. As a result, any PDE-based method can be 
applied to fill in a small, non-textured target region. 
There are several types of diffusion, including linear, 
nonlinear, isotropic (diffusion is the same in all 
directions), and anisotropic (diffusion changes depending 
on the object's orientation). 
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Figure 3. Illustration of Diffusion-Based Inpainting: 
2.2 Exampler based Texture Synthesis 


Many definitions and typologies of textures 
have been offered during the past four decades in digital 
image processing. An early method of image inpainting 
was based on algorithms that synthesize textures. Using 
patching techniques, the broken pixel is replaced with 
one from a nearby area that is similar to fill in the gap left 
by the damaged one. Figure 4 shows how texture 
generation adds additional pixels from an input taken 
from a known location and preserves the image structure. 
Most inpainting techniques use this technique, which 
involves copying and duplicating patches from existing 
image data to fill in the blanks where a matching patch 
does not exist. However, nature would be complete if the 
missing section were filled in with a similar visual look. 
Texture synthesis is more effective [3-4]. 


Synthesis algorithm 


Input a XS] 
iP 5 oO ‘ 122 1S 

texture wee PS RIEES 
Synthesized texture 


Figure 4. Illustration of Exampler based Texture 
Synthesis 
2.2.1 Typology of texture synthesis methods 


Over the last few decades, research in texture 
synthesis has resulted in the creation of a wide range of 
synthesis methods. These methodologies can be divided 
into three broad groups: procedural, exemplar-based, and 


model-based texture creation strategies. 


Texture synthesis 
methods 


Procedural synthesis Exem plar-based Model-based texture 
methods texture synthesis synthesis 
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Figure 5. Classification of Texture Synthesis Methods 


Procedural synthesis methods : Methods of procedural 
texture synthesis involve the creation of textures through 
the application of mathematical functions or algorithms 
that can be carried out with a predetermined amount of 
computational expense. As a result, procedural methods 
are tools that are ideally suited to producing the textures 
of objects in virtual environments, such as video games. 
Exemplar-based texture synthesis: In order to function, 
these methods require the input of one or more example 
textures. The majority of them generate textures by 
directly copying the pixels or patches (sub-images) from 
the images fed into the program. Therefore, the purpose 
of these methods is to produce a new texture designed to 
be as comparable to the model (input) texture or 
exemplar as is humanly possible. 

Model-based texture synthesis: These techniques involve 
building a probabilistic model capable of being used in 
describing the texture and synthesizing it. The model's 
parameters ought to capture the important observable 
qualities of the texture. 

2.3 Exemplar-Based Inpainting 


Texture synthesis techniques eventually led to 
the development of inpainting based on examples. It's 
difficult to fill in large textures with diffusion 
approaches. However, the example makes filling in large 
target regions easier by utilizing two distinct processes: 
1) picking the best matching patch and 2) performing 
priority assignments. Filling orders are based on priority 
and are determined by the exemplar, which aids in 
selecting the best matching patch from source regions 
(i.e., the regions closest to the target region). It is possible 
to discriminate between structure and texture by 
determining the pixel with the highest priority for each 
patch that has been briefly illustrated in [5], [6], and [7]. 
The missing part is then pixelwise reconstructed with the 
linear combination of these patches. Finding the best 
candidate is critical because it ensures a good match 
between the source and target regions. Finding the best 
candidate According to the filling order, a pixel from the 
closest neighboring patch is copied and filled into the 
target region. It seeks out the most comparable patch and 
duplicates the pixel from that nearby patch. The 
following are the steps for example-based inpainting: The 
exemplar-based image inpainting algorithm is a patch- 
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based approach that restores target regions in the input 
image by using these steps. 

1. Extract the target regions from the provided 
input image. 

2. Produce a binary mask with dimensions 
identical to the input images. The portions of the 
target image to be inpainted must correspond to 
the nonzero pixels in the mask image. 

3. Identify the region that is the source. The source 
region consists of all of the regions in the input 
image, except for the regions intended to be the 
targets. In other words, the source region equals 
the input picture minus the target regions. 

4. Using either the gradient or the tensor technique, 
determine the patch priority for each patch, each 
of which has dimensions of p by s and is 
centered on a boundary pixel in the target 
region. 

5. Identify the repair that has the highest priority. 
This particular patch will serve as the target for 
the subsequent inpainting. 

6. Using the sum of square differences, search the 
source region for the patch closest to the target 
patch you have (SSD). 

7. Transfer the image data from the patch that 
closely matches the destination patch. 

8. Ensure that the input image, mask, and patch 
priority value are all updated. 

9. Continue to repeat steps 4-8 until the desired 
areas have been inpainted. 


Figure 6(a) Extract the target regions 


| Mask for Inpainting 


= 


e 3 


Figure 6(b) Binary Mask 
. meu. 


Figure 6(c) Inpainted Image 


Figure 6. Illustration of Exemplar-Based Inpainting 
2.4 Hierarchical Super-Resolution Based Inpainting 


This adds exemplar-based inpainting — that 
combines inpainting and super-resolution. This method 
converts a high-resolution image to a low-resolution 
image, then inpaints it. This simplifies both the 
computational and visual aspects of working with image 
structures. First, pick the editing area in Hierarchical 
Super-resolution. A selected area within the limit will be 
colored. Convert high-resolution images to low- 
resolution. The low-resolution input image is broken into 
small patches and inpainted one by one. The resulting 
image is turned into a high-resolution image through a 
single-image super-resolution technique. 

This methodology was introduced by Olivier Le 
Meur et al. [8] for exemplar-based inpainting. This 
method is extensively used, although it has many 
drawbacks, the most prominent of which is a lack of 
control over patch size and order. First, an input image is 
inpainted using an exemplar before applying hierarchical 
super-resolution. In this case, the low resolution is 
incorporated multiple times. The results are quickly 
integrated and turned into a single image by utilizing the 
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super-resolution process via loopy belief propagation. 
Using this method, high-resolution details can be 
reconstructed quickly and accurately. The primary 
method used is extracting high-resolution patches from 
an image's recognized regions. It is easier to paint low- 
resolution images than high-resolution ones. The 
method's core principle is that it is based on the idea of 

1) The original image is re-sampled at a lower resolution 


to produce the final image. 

2) A low-resolution image has defects, so an inpainting 
algorithm is used to fix it. This algorithm employs 
dictionary construction and similarity computing to fill in 
the holes. 

3) The quality of the image is increased by developing a 
higher resolution from the image that was inpainted to 


recover details from the areas that were removed. 


Figure 7(b) Super-resolution- 


Figure 7(a) Mask 


based inpainting. 


first image would be processed by an algorithm for 
structural inpainting [8], while an algorithm would 
handle the second image for texture synthesis proposed 
by Efros and Leung [9]. Both of these algorithms would 
be applied to the respective images. Both of these 
processes leave their mark on the finished product 
somehow. Atzori and de Natale [12] came up with a 
suggestion for an additional hybrid technique. In this 
particular instance, the process of restoration begins with 
matching the contours that intersect the edge of the 
occluded area that is located inside the object. This 
process will result in smaller sections, which will be 
filled by copying blocks from the data perimeter. Rares et 
al. [11] recommended that while combining the edge 
contours that overlap the damaged region, both the local 
and global information should be considered. This was 
one of their recommendations. In this way, pairs of more 
precise lines will be obtained, but matching them will 
become more complex. The values of the restored pixels 
are then assigned based on the pixels located close to the 
newly obtained contours and the pixels located along the 
edges of the occluded area. 


2.5 Hybrid Methods 


It is a method of image modification where 
missing parts of an image are estimated and replaced 
with available or external data. It combines the 
advantages of both diffusion-based and enhanced 
exemplar inpainting methods. The image's structure is 
treated with a diffusion-based approach, followed by 
adaptive patch size—based inpainting that makes the 
Hybrid Method. One of the hybrid inpainting methods 
was developed by Bertalmio et al. [10], who created an 
algorithm based on the concept of decomposing the 
original image into two layers. This method is considered 
to be one of the hybrid inpainting methods. The textural 
qualities should go in the second layer, whereas the 
structural characteristics should go in the first layer. The 
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Input Image 


Structure (Geometry) 


Textures 


Figure 8(a) Illustrations of Hybrid methods 
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Table 1. Traditional Image Inpainting Techniques Overview 


Techniques 


Inference 


Future Work 


Diffusion-based 


e It generates good results when filling in small 


e To extend this approach for filling in large 


image in or gap regions. holes 
painting e 
Blur effect, which becomes obvious when | e Fails to inpaint large textured regions, 
filling larger regions resulting in blurry artifacts on image. 
e Preserves edge information. 
e 
Suitable for completing lines and curves. 
e 
Does not produce verbatim copies in the 
synthesized textured region. 
e 
Maintains the structure of the inpainted 
region. 
Texture synthesis | Similar patch doesn’t exist while synthe | Extensions include accurate Propagation of 
and sizing regions and not designed to handle | curved structures in still photograph and use of 


Exemplar-based 
image 


curved structures 


inpainting in videos. 


inpainting 

Exemplar based | By utilizing image information from multi | Alternate weighing functions and methods and 
image in | ple samples helps in improving visual qual | can be used for selecting samples. 

painting ity 


Hierarchical Super 
Reso 


Experimental results on a wide variety of 
images have demonstrated the effectiveness 


More images can be used for building dictionary 
to be used in super resolution 


lution of this method 

Sparse e Inpaints facial images with high exposure to | e May not work well on natural scene images. 
representation light. e On image reconstruction, the damages pixels. 
Methods e Allows change in light intensity. | e Must be in the vertical direction. 

Hybrid e Preserves edge and restores smoothness. | e Ifthe damaged area is large and 


Impressive results on the linear structure of 


the image improve the speed. 
e Smoothness iS restored 
and structure and texture are 


preserved of image. 


inappropriate, blocky image is obtained 
e Computational complexity with no guarantee 
in convergence. 
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3. Deep Learning Methods 


Recent work [13] summarized the image 
inpainting technology based on deep learning. Omar 
Elharrouss et al. [13] divided the image inpainting model 
methods proposed in some classic papers into three 
categories from a global perspective, namely, sequence- 
based methods, CNN-based methods and GAN-based 
methods. Qiang et al. [14] summarized the main image 
inpainting methods based on deep learning in recent 
years and classified the existing methods into three 
network structure types of image inpainting methods 
based on convolutional autoencoder network, generative 
adversarial network and recurrent neural network 
according to the inpainting network structure. 

Image inpainting [14] is a research field of 
image processing. It aims to fill the missing or masked 
regions of the image with generated content and make the 
repaired image visually realistic. Image inpainting 
technology has been widely applicated in many fields, 
including ancient book restoration, medical image 
processing, and PhotoShop processing. Therefore, the 
research of image inpainting is worth studying. Due to 
the complexity of the natural images, there will be 
obvious fuzzy phenomena in the region of the repaired 
image and the boundary between the original region and 
the repaired region, which is a main difficult issue in the 
work of image inpainting. Also, how to ensure the 
semantic correctness of the repaired region is one of the 
difficulties in the task of image inpainting. In order to 
address these problems of image inpainting, existing 
methods categorize into two types: one category is 
texture synthesis methods based on the patch, the main 
idea is to find the boundary of the missing region to fill in 
the missing part of the image. 

3.1 CNN based Technique: 


CNN has proved powerful in many tasks, 
including single image inpainting [20], the main idea is to 
extract the features of the image through the deep 
convolution neural network to understand the image, and 
then to fill the missing region. Deep neural networks 
have been successfully applied to problems such as 
image segmentation, image super-resolution, coloration 
and image inpainting. Several methods have been 
proposed for image inpainting using convolutional neural 
networks (CNNs) or encoder-decoder network based on 
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CNN.A typical patch-based method is a Patch-Match 
method proposed by Barnes et al. [15], which searches 
for the matching patch from the rest part of the image to 
fill in the missing region, resulting in more reasonable 
texture information. This method has an excellent effect 
on background inpainting. However, it does not perform 
well in the face of complex images (face, natural images) 
inpainting, and the result of inpainting will be very 
vague. Similarly, other patch-based methods [17] [18] 
and exemplar-based methods [16] [21] are weak in 
inpainting the missing regions with complex structures. 
The reason is that the texture synthesis method based on 
the patch is still not enough to obtain the high-level 
characteristics of the image. With the rapid development 
of deep learning, the appearance of the feature learning- 
based image inpainting method exactly fills the defect of 
the traditional image inpainting methods, which is lack of 
high-level coherence and difficult to deal with the 
problem of large areas or complex structures missing. 
Neural networks are more powerful to learn high-level 
semantic information of images and CNNs are effective 
tools for image processing [20]. Neural networks do 
more and more image inpainting tasks. Pathak et al. [30] 
proposed the model Context Encoder, which combines 
encoder-decoder and Generative 
Network(GAN) [19] to train in an unsupervised method. 
They use the adversarial loss to make the repaired image 
as real as possible and generated realistic results. But 


Adversarial 


context encoder has drawbacks: the fully connected layer 
cannot save accurate spatial information and context 
encoder sometimes creates blurry textures inconsistent 
with surrounding areas of the image. 

3.2 Generative Adversarial Network (GAN) 


Generative Adversarial Networks (GANs) are a 
powerful class of neural networks that are used 
for unsupervised learning. It was developed and 
introduced by Ian J. Goodfellow in 2014. 

In GANs, there is a generator and 
a discriminator. The Generator generates fake samples of 
data(be it an image, audio, etc.) and tries to fool the 
Discriminator. The Discriminator, on the other hand, tries 
to distinguish between the real and fake samples. The 
Generator and the Discriminator are both Neural 
Networks and they both run in competition with each 
other in the training phase. The steps are repeated several 
times and in this, the Generator and Discriminator get 
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better and better in their respective jobs after each 
repetition. The working can be visualized by the diagram 
given below as shown in figure 9: 

Here, the generative model captures the 
distribution of data and is trained in such a manner that it 
tries to maximize the probability of the Discriminator in 
making a mistake. The Discriminator, on the other hand, 
is based on a model that estimates the probability that the 
sample that it got is received from the training data and 
not from the Generator. 

The GANs are formulated as a minimax game, 
where the Discriminator is trying to minimize its 
reward V(D, G) and the Generator is trying to minimize 
the Discriminator’s reward or in other words, maximize 


Real 
samples 


Generator 


Generated 
fake samples 


its loss. It can be mathematically described by the 
formula below: 

min max V(D, G) 

G D 


V(D, G) = Exnpsata(z) logD(x)] + E,~p, (2) llog(1 — D(G 


Where G = Generator 
D = Discriminator 
Pdata(x) = distribution of real data 
P(z) = distribution of generator 
x = sample from Pdata(x) 
z = sample from P(z) 
D(x) = Discriminator network 
G(z) = Generator network 


Condition 


bs it correct? 


Fine tune training 


Latent random variable 


Figure 9. Flow model of Generative Adversarial Network 


Generator 


Inpainting is part of a large set of image 
generation problems. To solve this problem, we use an 
auto-encoder as the generator of our model. The auto- 
encoder contains two networks: an encoder and a 
decoder. In this paper, we input the image to be repaired 
into the encoder and encode it into code, and then 
reconstruct and generate the repaired image via decoder 
decoding. 

Discriminator 


The generator is responsible for inpainting the 
missing or masked regions of the image. However, the 
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Generator cannot guarantee that the generated regions are 
accurate or consistent with the original image. In order to 
ensure that the generated image is much more realistic, 
this paper uses the discriminator as a binary classifier to 
distinguish whether the image comes from real data 
distribution or generated by the generator. Also, the 
discriminator helps to improve the generator’s ability to 
generate more realistic images to fool the discriminator. 
Deep learning-based image inpainting approaches that 
can generate the missing pixels in an image with good 
global consistency and local fine textures. 
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Table 2. Deep learning-based inpainting approaches 


Approach 


Significance 


Advantages 


Disadvantages 


Context Encoder (CE)[22] 


first Generative Adversarial 
Networks (GANs) [23] based 
inpainting algorithm 


* Doesn’t require 
high memory usage 
«Simple 

*Performs well for 
center square missing 
regions 


e Fails if the location of 
the missing region 
changed 


e Does not support 
random missing regions 


Multi-Scale Neural Patch | Enhanced version of CE Performs well for | Slow algorithm 

Synthesis (MSNPS, 2016) center missing region | Does not 

[24] support random 
missing regions 
Fails when the 
image is not 
center symmetric 
images 

Globally and Locally | It is a milestone in deep image | ¢ Performs well Does not support 

Consistent Image | inpainting as it defines the Fully | « Efficient random missing 


Completion (GLCIC, 2017) 
[25] 


Convolution Network — with 
Dilated Convolutions for deep 
image inpainting 


regions 
* Unable to handle 
complex structures 


Patch-based Image 
Inpainting with GANs [26] 


two advanced 
namely residual _ learning [27] 
and PatchGAN [28] were 
embedded in GLCIC to further 


concepts 


+ Efficient 

¢ Performs well for 
center square missing 
regions 


¢ Added artifacts 
¢ Fails in the case 
of images with 

different textures 


boost its inpainting and colors 
performance. 
Shift-Net [29] takes both the advantages of | Shift-Net inherits Guidance loss _ that 


modern data-driven CNNs and 
the conventional “Copy-and- 
Paste” method in the form of 
Deep Feature Rearrangement by 
using the proposed _ shift- 
connection layer. 


the advantages of 
exemplar-based and 
CNN-based methods, 


encourages the decoded 
features of the missing 
parts (given a masked 
image) to be close to 
the encoded features of 
the missing parts (given 
a good-conditioned 
image). 


Shift-connected layer 
with shift operation 
allows the network to 
effectively borrow the 
information given by 
the nearest neighbours 


outside the missing 
parts to refine both the 
global semantic 
structure and local 


texture details of the 
generated parts. 
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Generative Image Inpainting 
with Contextual 
Attention (CA, 2018)[30] 


Can be regarded as an enhanced 
version or a variant of Shift-Net 
[31]. 


¢ Performs well 

for fixed square 
missing region 
*Supports 
highresolution images 


* Does not support 
random shaped missing 
region 


Generative Multi-column 
Convolutional Neural 
Networks (GMCNN, 2018) 
[32] 


The multi-column _ structure 


method 


It can decompose 
images into 
components with 
different receptive 
fields and feature 


Expands the 
importance of 
sufficient receptive 
fields for image 


inpainting and proposes 


resolutions. new loss functions to 
further enhance local 
texture details of the 
generated content. 
DeepFill v2 (A Practical | This can be regarded as an | achieves high-quality | Gated Convolution 


Generative Image Inpainting 
Approach, 2019) 


enhanced version of DeepFill v1 
[9], Partial Convolution [33], 
and EdgeConnect [34]. 


free-form inpainting 
than previous state-of- 
the-art methods 


which is a_ learnable 
version of the Partial 
Convolution. By 
adding an extra 
standard convolutional 
layer followed by a 
sigmoid function, the 


validness of each 
pixel/feature location 
can be learned and 
hence optional user 
sketch input is also 
allowed. 
A Novel Face Inpainting | A new novel approach for face | ¢ Efficient * Fails to produce good 
Approach Based on Guided | inpainting is proposed which | « Preserve human results if the human 
Deep Learning can capture and preserve the | identity in images face is not centered in 
identity of each human face in | * Works with random | images 
images while reproducing the | shaped missing | * Does not support 
missing irregular region regions highresolution images 


images 


4. Image Inpainting Challenges 


After detailed investigation on Image inpainting with 
several traditional and deep learning methodologies we 
noticed some of the research challenges. 

Challenge 1: Research is required on 2D image 
inpainting and depth map inpainting 

Challenge2: working with structural, textural or both the 
methods simultaneously for image inpainting . 

Challenge 3: There are no current methods that can 
reliably inpaint large holes in depth map. 

Challenge 4 : Depth inpainting is still a challenging issue 
for medical image application . 
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5. Conclusion 


Computer vision applications rely on image 
inpainting to perform various data modifications, 
including image quality enhancement, restoration, etc. A 
brief survey of image inpainting is provided in this paper. 
Traditional and deep learning algorithms are among the 
options that have been discussed. Different approaches to 
traditional and deep learning were discussed here. None 
of the methods studied can correct for every sort of 
distortion in an image. However, the results obtained by 
utilizing deep learning are pretty encouraging for each 
category examined. On the other hand, deep learning has 
difficulty with its high computational complexity and 
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memory requirements. Researchers must focus more on 
developing algorithms that can deal with both simple and 
complicated structures in the future. Computer vision 
applications rely heavily on image inpainting because of 
the vast amounts of data that can be altered utilizing 
image editing tools. These applications include wireless 
image coding and _ transmission, image quality 
improvement, image restoration, etc. An introductory 
survey of image inpainting is offered in this paper, 1.e., 
sequential (non-learning) techniques, _CNN-based 
approaches, and GAN-based approaches are just a few of 
the ways discussed. In addition, we noticed a few 
research challenges in attempting to compile various 
methods for dealing with various potential future 
suggestions. 
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